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ABSIBACT 

This paper addresses the use o£ multiple regression 
in th« analysis of reading and math scores. The data presented\in the 
paper were from the Mesa Public Schools in Arizona, and Here conpdled 
during the 1975-76 school year. Five criterion factors were selected 
for investigation: {%) third grade Stanford reading scores, (2) third 
grade criterion reading scores, (3). third grade criterion math 
scores, (4) fifth grade Stanford math scores, and (5) sixth grade 
criterion math scores. The criterion reading and math scores were for 
the Mesa-developed criterion reading and math tests. Other data was 
then summarized to form eleven demographic factors that could be used 
to predict the value of these criterion variables for each of the 
schools for the 1975-76 school year. Through the use of multiple 
regression, predicted or expected values were determined for 
individual schools. These values were then- used to--compare with the 
actual scores the school attained. Through the use of multiple 
regression an individualized predicted value or mean value was 
determined for each school based on its demographic characteristic. 
Comparisons of achievement of the school were then made based on its 
individual characteristics. This method was then contrasted with 
simple comparisons of a school's achievement with the district mean 
value and national norm mean. (Author/MV) 
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Foreword 

The Mesa Public School District covers grades K through. 
12. It serves approximately 30^000 students with 25 elementary 
schools, 6 junior high schools and 4 senior high schools. 

It is the major function of the Mesa Public Schools Depart- 
ment of Research and Evaluation to provide information to 
decision makers. This information should be timely, valid, 
reliable, and of sufficient depth, yet concise enough to speak 
squarely to the issues, at hand. The decision makers noted 
above range from the classroom teacher to the superintendent 
of the district, depending upon the need for specific informa- 
tion. Persuant to the accomplisihment of this task, the Depart- 
ment has developed what they consider a viable method of comparing 
achievement in basic skills at the unit level. This report then 
is the type of report that is furnished to- district administra- 
tors so that they in turn can develop specific objectives at the 
unit level based on as much information as possible. 

Introduction ^ v 

One of the major problems in determining if a given basic 
skill program at a school is performing up to expectation is 
in determining what yardstick is available for measuring the 
achievement in the basic skill area, A number of yardsticks 
or methods are certainly available and used. One is the use 
of longitudinal data to determine program effectiveness. -Al- 
though this type of information can be used to determine changes 
over years, assuming the populations have remained constant, 
the information cannot by itself be used to determine the effec- 
tiveness of a given program. One other method/ used is com- 
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paring the scores at a given school with a norm value. The 
drawbacks in this case are the same as those whenever norm- 
referenced data are used, that is, is the norm a realistic 
one for the particular district and the individual 'school that 
is being used in the comparison? As can be seen from Figure 1, 
when comparing the elementary schools in the Mesa district to 
the norm mean all but two schools equal or exceed the norm. 
This could lead to the conclusion, which may be erroneous, that 
everything is fine in River City. The students in the Mesa 
district, however, have above average I.Q. scores, the median 
income of the families are above average and the percent of 
minorities is lower than the national average. If these and 
other factors are incorporated in the analysis, one must question 
the advisability of resting on the laurels of the schools' 
reading scores. Another method to determine program effective- 
ness is to set or determine a given criterion for a particular 
school. This method also has drawbacks , the* main ones being 
the difficulty of 'determining an individual school criterion 
and the lack many times of objectivity in the task of setting 
realistic objectives for determining program effectiveness at 

an individual school. - - — ^ — - 

Still another method would .be-.±o-^use>--theL-.dis±rict_..mean_as 

the standard and compare all the schools' achievement with* the 
mean score plus or minus some tolerance on this particular 
achievement factor. Figure 1 shows what happens when this 
method is used. The mean results of the third grade Stanford 
reading test are presented by school. The tolerance was deter- 
m^ined by going three standard errors above and below the mean; 
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Figure 1 



Mean Third Grade Stanford Reading Scores by School 
with Corresponding Tolerance Interval About District Mean 
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that is, tolerance limits of 2.95 to 3.33 about^a' mean of 3.14. 
As can be seen, 7 of the 25 schools fall within the range 
about the mean with 8 above 'and 10 below the range. Thus, 
it would appear that student achievement in the Mesa district 
is at least satisfactory for 60% of the schools. Here again, 
as in the example where a national norm value was used, there 
is not sufficient information to judge that the reading program 
at Washington is outstanding, that at Hale satisfactory and 
that at Franklin poor. To really consider the programs at 
these schools one must consider that the mobility rate at 
Frankling is over 50% during the year, i.e., more than 50% of 
the students at Franklin are new to the school during the school 
year. The absence rate at Franklin is among the highest in the 
district. The converse of these factors is true for Washington 
with Hale somewhere in between. When one considers the reading 
scores compared to the district mean, therefore, is this a good 
yardstick for measuring program effectiveness? 

It is not being said that the above examples of the use. of 
data or information are not valid and useful methods of using 
the information. What is being said is that there is a need for 
a new yardstick for comparing achievement in the basic skill 

areas_.-^^. One that attempts to overcome the .deficiencies noted 

in the methods previously discussed. This is not to say that 
one method can overcome all deficiencies, but it is apparent 
that a need exists for an additional yardstick to fill the gap 
in the current measuring devices. This paper, then, addresses 
this need and proposes the use of multiple regression techniques 
in developing such a yardstick. 
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Method 

One of the major . problems in determining if a given pro- 
gram at a school is performing up to expectation is in finding 
comparable data on which to base the decision. In this docu- 
ment, an attempt is made to furnish the building administrators 
with data that can be used to make this kind of judgment* 

Six criterion factors were selected for the investigation. 
These factors are: third grade Stanford reading scores, fifth 
grade Stanford reading scores, third grade criterion reading 
and math scores, and sixth grade criterion reading and math 
scores. Data were then collected on factors that could be used 
to predict the values of these factors for each school for the 
1975-76 school year. The values of these factors can be found 
in the Mesa School Profile 1975-76. Step-wise multiple regres- 
sion techniques:. were then used to determine which were the best 
prediction factors (see Appendix). These factors were then 
used to come up with a predicted value or an expected value 
for a given school. The prediction equations take into account 
the selected demographic factors or prediction factors in 
arriving at the predicted value. The building administrator 
can then use these predicted values to compare to the actual 
values for last year. 

The "definition of the different criterion and prediction 

factors that were used in this study are given below. 

Criterion Factors . , 

1. Third Grade Stanford This is measured by the average 

Reading third- grade reading score achieved 

on the Stanford given in October 
1975-76. 



2. Fifth Grade Stanford This is measured by the average 

Mathematics fifth grade math score achieved 

on the Stanford given in October 
i 1975-76. : 



Third Grade Criterion 
Reading 



Third Grade Criterion 
Mathematics 



Sixth Grade Criterion 
Reading 



Sixth Grade Criterion 
Mathematics 



Prediction Factors 



1. School Type 



2. Student Inward 
Mobility Rate 



3. Ethnicity 



Student Intellectual 
Level 



Expenditure per 
Studexit 
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This is measured by the average 
percent correct attained by the 
third grade students on the district 
criterion-referenced reading test 
administered in May. 

This is measured by the average 
percent correct attained by the 
third grade students on the district 
cr iterion-ref erenced math test ad- 
ministered in May, 

This is measured by the average 
percent. correct attained by the 
sixth grade students on the district 
criter ion-referenced reading test 
administered in May. 

This is measured by the average 
percent correct attained by the 
sixth grade students on the district 
criterion-referenced math test ad- 
ministered in May. 



School type was simply coded by 
type of school.- This factor was 
included more for prediction of 
plant vandalism then cognitive 
scores. The newer plant designs 
are more vandalism preventative. 

This was measured by taking the 
aggregate total of the new stu- 
dents during the school year and 
dividing that by the" aggregate 
total of all students at that 
specific school during that year. 

This was measured, by taking the 
total number of Anglo students 
at the end of the school year and 
dividing by the total number of 
students at the' end of the school 
year . 

This is measured by the average 
second grade score on the Otis- 
Lennon test given in October 1975-^76 

This was measured by the total 
teacher expenditure plus the ex- 
penditure for books and supplies - 
for a given school divided by 
the student population- at that 
school. ' 



6. Student Absence Rate This was measured by taking the 

total number of student absences 
at the school and" dividing by ^ 
the total student population at 
the given school, 

7. Teacher Academic Level This was measured by taking the 

total teacher population at a 
given school at the end of the 
school year and dividing it into 
the total number of teachers v/ith 
only bachelors degrees. 



8. Teacher Experience 
Level 



This was measured by taking the 
total number of teachers with over 
6 years of experience at the end 
of the school year and dividing 
by the total number of teachers 
at the given school. 



9. Teacher Ethnicity . This was measured by taking the 

total number of Anglo teachers 
at the end of the school year and 
dividing by the total number of 
teachers at the end of the school 
._year. 

10. Teacher Absence Rate This was measured by taking the 

total number of teacher days absent 
. and dividing by the total number 
of teachers at the given school. 

11. Percent of Openness * This was measured by taking the 

estimated physical percent of 
openness at each school . 

In addition to the predicted values for each school the 

standard error of these estimates were determined. 



Results , 

Table I presents the actual values for each of the cognitive 
criterion factors along with the corresponding determined pre- 
dicted values for the factors. Those cases in which the actual 
value fell below the predicted value by 3 times the standard error 
of the estimate are highlighted. In the case of the third grade 
criterion ^math, none of the prediction factors were found to . 
have high predictability. The best estimate of an expected 
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Table I 

Actual versus Predicted Values 
for Selected Cognitive Criterion Factors 





3rd Grade 
Crit, Math 


3rd Grade 
Crit. Rdg. 


Stan . 
Gr . 


3rd 
Rdg. 


Stan . 
Gr . 


5th 
Math 


6th C 
Crit 


Srade 
Math 


6th C 
Crit 


Jrade 
Rdg . 


School 




Pred . 


Ajct . 


Pred . 


Act . 


Pred. 


Act . 


Pred. 


Act. 


Pred. 


Act. 


Pred. 


Adams 


87 


80 


86 


82 






5.2 


5.1 


iii 




86 


89 


Edison 


80 


80 


86 


88 


3.4 


3.3 


,5.5 


5.6 


76 


71 


92 


89 


Eisenhower 






IS 




3.3 


3.2 


5.4 


5.2 


66 


69 


86 


87 ^ 


Emerson 


76 


80 






3.5 


3.3 


5.5 


"5.3 


79 


71 


94 


90 


Field 


82 


80 


84 


82 


lilBi 




5.3 


5.1 


67 


68 


88 


89 


Franklin 


84 


80 


83 


81 


2.7 


2.8 


4.8 


4 . 8 


63 


62 


85 


83 


Hale 


85 


80 


84 


83 






5.3 


5.2 


70 


69 


88 


89 


Hawthorne 


87 


80 


89 


85 


3.8 


3.4 


5.6 


5.6 


70 


72 


90 


89 


Holmes 


88 


80 


84 


85 


2.9 


3.0 


4.9 


4.9 






8 6. 


86 


Jefferson 


79 


80 






2.9 


2.9 






66 


66 


88 


87 


Keller 


82 


80 


86 


82 


2.9 


2.6 


4.9 


4.5 


78 


62 


90 


86 


Lehi 






fill 








isi 












Lincoln 


;$?® 




80 


82 


3.0 


3 . 0 






66 


66 


89 


87 


Lindbergh 


93 


80 


91 


82 


3.5 


3.3 






76 


68 


91 


8 9 


Longfellow 






81 


83 


3.4 


3.1 


5.4 


5.2 


73 


67 


90 


87 


Lowell 


84 


80 


81 


80 


3.2 


3.2 


5.1 


5.2 


68 


71 


89 


90 


Redbird 


79 


80 


81 


80 


3.2 


3.1 


4.9 


4.7 


65 


68 . 


92 


90 


Roosevelt 


84 


80 


88 


87 


3.5 


3.5 


5.8 


5.7 


79 


74 


95 


91 


Salk 


80 


80 - 


80, 


80 


2.9 


3.0 


im 




:0\';r:'-l'it-'<s' 




87 


88 


Stevenson 










2.6 


2.7 


4.7 


4.6 






fsbp 




Taf t 


84 


80 


81 


76 


3.0 


3.0 




Sill 


68 


65 


87 


87 


Washington 


82 


80 


86 


83 


3.4 


3.5. 


5.5 


5.6 


72 


72 , 


jS" 


PI 


Webster 


78 


,80 








ill 


5.4 


5.3 


67 


69 






Whitman 


90 


8 0 


89 


85 ■ 


3.1 


3.2 






66 


68 






Whittier 


WM 




85 


86 


3.7 


3.4 


5.8 


5.4 


^%5;S 
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value is, therefore, the district mean which is presented in 
Table I. ' ■ ' , 

To illustrate this method further, Figure 2 presents a 
graphic example of the results. Third grade mean Stanford 
reading scores are presented by school. The expected or pre- 
dicted values are also plotted with corresponding tolerance 
intervals. By comparing Figure 1 with Figure 2 the difference 
in the results of the use of different yardsticks is quite 
evident. For example, Hale School was reading significantly 
above the national norm and well within the tolerance limits 
about the district mean. When the demographic characteristics 
of the Hale students are taken into account, however, and a 
predicted value for the Stanford third grade reading scores 
was determined, it can be seen from Table I and Fiigure 2 
that Hale School is in fact reading below their predicted 
value and significantly below. By contrast. Franklin School, 
which when using the two other yardsticks was reading below the 
national norm and significantly below the district mean, was 
within its individual tolerance level xvhen the demographic 
characteristics of the students were taken into account through 
the use of a multiple regression framework. The third school, 
Washington/ which was used in the previous example was "reading, 
34, far above the national norm of 2.8, significantly above the 
district mean of 3.1; but when the demographic characteristics 
of the students were incorporated in the analysis, it was found 
that the students at Washington were reading below what would 
be expected of them. The value was still within the individual 
tolerance level, but it certainly does net show the outstanding 



3.8 



3.6 



3.4 



3.2 



3.0 



2.8 



2.6 



2.4 




Figure 2 

Mesa Third Grade Stanford Reading Scores by School 
with Corresponding Individual Tolerance -Int-er-va-ts- 



"A Predicted Value 
# Actual Value 
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performance that was evidenced when compared to the national 
norm or district mean. Similar procedures and analyses are , . 
used for each of the criterion factors when analyzing the pro- 
grams for each of the individual schoo"^ • 

Carrying the analysis one ste: ^ lool achievement 

versus their potential as determineu ^ tr..o predicted values 
were summed over the six criterion factors • A simple plus 
minus scheme was used in this analysis. The. d • i ations , 
being coded simpTir'¥s'^~plus 1 or a minus 1, e summed over 
the six criterion variables and the schools were grouped 
according to: 

Schools performing above potential 

Schools performing at potential 

Schools performing below potential 
Table II presents the results of this analysis. 

Summary and Conclusions 

The Mesa Public Schools have been using this yardstick for 
comparing achievement in the basic skill areas for the last 
three' years. It certainly is not used in isolation, but is one 
more added bit of information furnished to the superintendency 
so that they in turn can work with the unit level administrators 
in setting up specific goals and objectives at the unit level. 
Other measures are also furnished, to the superintendency. The 
district mean is taken into account, the national norm value 
is considered, longitudinal data is examined, but when all is 
said and done the demographic characteristics of the students 
or the clientele that is actually served at the school must be 
taken into account when examining program effectiveness at 
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TABLE II 

School. Achievement versus Potential 



Schools Performing 


XiQioon 


Above Potential 


Emerson = 


(Sum of deviation 


Franklin " 


direction 


Hale 


+6 to +2) 


Hawthorn 




Keller 




Lindbexv^li 


■■ 


Longfellow 




ReaDira 




Roosevelt 


...... ^ 


Taft 


Q r»h r\r\ 1 c: P <=> y" "Fri nn "i no 


Adams -""^ 


at Potential 


Field 


^ o uUU (J i. Lie V J^Cl L.X ^^ii 




direction 


Jefferson 


"T/L uO x; 


Ti T_nco 1 n 




Lowell 




Washington 


Schools Performing 


Eisenhower 


Below Potential 


Salk 


(Sum of deviation 


.Stevenson 


direction 


Webster 


-2 to -^7) 


Whitman 


Whittier 




Lehi 
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this level. It is* felt that through the use of multiple 
regression this is done in a systematic manner, thereby in- 
suring the preclusion of subjective bias. 

The district feels that although this is a very useful 
and vvable technique, it is certainly not the end. The multipl 
regression techniques used to determine the predicted values 
are based on school values i.. 1" on individual student values 
It is felt that a more vaii reliable measure will come 
with the use of individual student data. School predicted 
values will then be determined by weighting by the propor- 
tional number of students with those demographic character- 
istics at the individual school. Attempts are being made cur- 
rently to generate a viable student data base which can be 
used in such a process. Another problem area is the use of 
grade equivalency scores in the analysis of the Stanford third 
grade reading and fifth grade mathematics. It is realized 
that raw scores or standard scores would be better measures 
to use in the analysis, but Mesa, as is the case in other 
districts, has felt the pressure of using a variable that the 
general public seems to accept. Strides are also being made 
in this area to re-education the public to the use of other 
scores and the misuse of the grade equivalency- score , _ 
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ADDENDUM 

Table III presents the rankings of the schools on each 
of the six cognitive criterion factors. The rankings were deter- 
mined by finding the deviation of the actual scores from the 
predicted scores . In addition to the rankings given for each 
of the individual criterion factors, ^n overall ranking of the 
schools was determi t^'^^^I b; :>ununing across the runks of the Individ 
ual criterion fa^uoj. £> and reranking to determine \;hich school 
performed the best as measured by the 'deviation of actual score 
from the predicted score over the six criterion factors . In 
all cases, a rank of one was given to that school with the best 
performance. 

Table IV presents the overall rankings of the schools for 
the 1973-74, 1974-75 and 1975-76 school years. 
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TABLE III 

School Rank on Achievement Factors . • \ 



School 


3rd Grade 
Crit. Math 


3rd Grade 
Crit. Rdg. 


6th Grade |6th Grade 
Crit. MathJ^rit. Rdg. 


5th Grade 
Stan. Math 


3rd Grade 
Stan. Rdg. 


Over-J 
all \ 


(1 Adan\s 


4.5 


4 . 5 


22 


21.5 


10 


23.5 : 


17.5- i 



* CiWlJLovJll 


X *s • -J 


18 


5.5 


4.5 


17 


8 


9 . 5 < ^ 


< ill j^aciiriower 




9^ 


18 


16 


' 5 ■ 


8 


21 ' i 

-i 


s ciiuc^ 175 on 




90 


2.5 


2 


5 ".■ 


■ 5 . 5 ^" vi;^ 


5 ^5 1 


t; -1 . 

rie±a 




ft ^ 


13 


16 


■ , 5 ' ■ . 


'< ,23 . '5 


K 

12 : ? 


' Franklin 




ft 


8.5 


■ 7.5 




17.5 


7 ' ! 
■■| 


Hale 


b 




8.5 


16 


10 


21.5 


11 1 


" Hawthorne 






15 


10.5 


1 4 


1 


3.5 1 

' 1 


Holmes 


3 


15.:-. ' 


i 2:2 


12.5 


1 4 


175 


16 ^ 



V Jefferson 




PI " ' I 


1 


10.5 


20 5 


12 


19.5 \ 


Keller 


1^ 


4 . ^ ,1 


\ 1 


2 


1 5 




1 1 


. Leni 


Z / • D 


9 T ■ 


24 


19 


20 5 


21.5 


25 1 


>jjxnco±n 




1 ft ' 


11 


7.5 


20.5 


12 


19.5 J 

• ■ ■ ' \i; « 


-r: Lindbergh 




1 


2.5 


/ 7.5 


25 


5.5 


2 J 


■rr .Longfellow 


9 1 


1 ft 


4 


.4.5 


5 


3 


• • ■ - f 


Lowell 


O • 3 


11 ^ 


18 


16 


17 


12 


13 1 


' ' ■ ^ 1^ « «A ^ 

ROCLDirCl 




1 1 ^ 


18 


7.5 


5 


8 


8 1 


-^Koosevei t - 


,0.»-D. - 


11 ^ 


.5.5 L 


2 


10 


12 




' SaiK 




1 A 


20 


16 


20 5 


17 . 5 


22 


. Stevenson 


O C 


9 / 


2 5 


25 


10 


17.5 


24 ' f 






2 


1 7 


12.5 


2 3.5 


14 


9.5j 


' Washington 


12 


7 


rn — 


21. 5 


17 


17.5 


17.5 ' 


Webster 


18 


22 




24 


10 


25 


23 


Whitman 


2 




15 


21.5 


2 3.5 


17.5 


15 


Whittier 


20 


15.5 


22 


21. 5 


1.5 


■ • 3 . 


14 


i 0 ■ ■ ' " 

lERjlClfe;,:,,, 






r 


1 , , , 









TABLE IV 

Overall Rank on Achievement 
Factors for 1973-74, 1974-75 and 1975-76 



Schools 


1973-74 
Rank 


1974-75 
Rank 


1975-76 
Rank 


Adams 


3 


3 


17. 5 


Edison 


9 


11 


9 . 5 


Eisenhower 


11. b 


A 
H 


2 1 


Emerson 


20 




■ c 

5 


FieXa 


17 


2 J 


1 0 
12 


Franklin 


lb 


/r 
O 


•7 


Hale 


4 


c 
D 


11 


Hawthorne 


11. 5 


/ 


3 • 5 


Holmes 


o 1 
21 


1 T 

17 


1 /I 
16 


Jefferson 


8 


12 


19. 5 


Keller 






1 


Lehi 


1 f? 


o 1 
21 


25 


Lincoln 


19 


1.5 


19. 5 


Lindbergh 


23 


20 


2 


Longfellow 


1 




b 


Lowell 


22 


lb 


1 0 
13 


KeaDxra 


•7 

/ 


Id 


0 
8 


Roosevelt 


2 
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FACT0P5 USED IN PREDICTIONS 



Third Grade Criterion 
Reading Test 

Student Inward Mobility Rate 

Percent of Teachers with BA 



Multiple R 
( Accumulated) 

0.469 . 

0.575 



Sixth Grade Criterion 
Math Test 



Otis IQ 



(Accumulated) 
0.485 



Sixth GradB Criterion 
Reading Test 

Otis IQ 

Teacher Absence Eate 



Multiple R 
(Accumulated) 

0.493 

0.555 



Fifth Grade Stanford 
Math Test 

Otis IQ 

Teacher Absence Rate 
Percent of Anglo Students 



Multiple R 
(Accumulated) 

. 0.721 

6.790 

0.822 



Third Grade Startfbrd 
Reading Tesl: 

Otis IQ . 

S tudent . Aibsen ce Rate 



Multiple R 
(Accumulated) 

0.729 

0.773 



